Skip to content

Draft: Solve intermediate variable bug #19399

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 16 commits into
base: master
Choose a base branch
from
Draft

Draft: Solve intermediate variable bug #19399

wants to merge 16 commits into from
+238 −72

Conversation

randolf-scholz
Copy link
Contributor

@randolf-scholz randolf-scholz commented Jul 8, 2025 •
edited
Loading

Looking for feedback, as this was mostly a trial-and-error process.

  • Removed self.infer_function_type_arguments_using_context. Instead, we now use a function infer_constraints_from_context that only yields the constraints.
  • Modified infer_function_type_arguments to now include the context.
    We compute 2 sets of constraints: the "outer constraints", determined by the context, and the "inner constraints" determined by the callable type. From this, we compute 2 solutions:
    • the "outer solution", which only uses the outer constraints.
    • the "joint solution", which uses both the outer and inner constraints.
  • We prefer the "joint solution", unless:
    • it failed to solve some variables
    • the return type of the outer solution is a (strict) subtype (i.e. more precise) of the return type of the joint solution
    • the return type of the outer solution is a union, and any member of that union is a subtype of the joint return type.1
  • if the outer solution was selected, we essentially fall back to the original code from the master branch, and first apply the outer solution, and then recompute the inner constraints and apply the inner solution.

New Tests

Modified Tests

  • testRecursiveAliasWithRecursiveInstance: improved inference2
  • testCallerTupleVarArgsAndGenericCalleeVarArg now reveals int | None rather than Literal[1]? | None3
  • testInferenceAgainstGenericCallableGenericProtocol now reveals F[T | None] rather than def [T] (T|None) -> T|None4
  • testInferenceAgainstGenericParamSpecPopOn now reveals def (U) -> U rather than def [T] (T) -> T5

Footnotes

  1. This was needed to fix testLiteralAndGenericWithUnion, because the outer solution was int | Literal["foo"] whereas the joint solution was Literal["foo"]? which got converted into str later. ↩

  2. previously, a=b followed by a=[[b]] made a be inferred as __main__.B followed by builtins.list[Union[__main__.A, typing.Sequence[Union[__main__.A, ...]]]]. Now, it gets more precisely inferred as builtins.list[builtins.list[__main__.B]]. And afterwards, join(a,b) get more precisely inferred as typing.Sequence[typing.Sequence[__main__.B]] rather than typing.Sequence[Union[__main__.A, typing.Sequence[Union[__main__.A, ...]]]] previously. See: https://mypy-play.net/?mypy=latest&python=3.12&gist=619a7fa6c3ec9d75bd1492bf8e68432c. The new result is also more in line with the inference by pyright ↩

  3. same as pyright ↩

  4. same as pyright ↩

  5. same as pyright ↩

randolf-scholz
randolf-scholz commented Jul 8, 2025
View reviewed changes
randolf-scholz
randolf-scholz commented Jul 8, 2025
View reviewed changes
mypy/checkexpr.py
lambda i: self.accept(args[i]),
)

# ??? QUESTION: Do we need to recompute arg_types and pass1_args here???
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It seems to work both with/without recomputing arg_types here, but I am not sure.

@randolf-scholz randolf-scholz mentioned this pull request Jul 8, 2025
Open
randolf-scholz
randolf-scholz commented Jul 9, 2025
View reviewed changes
mypy/checkexpr.py
Comment on lines +2111 to +2125
# HACK: convert "Literal?" constraints to their non-literal versions.
inner_constraints: list[Constraint] = []
for constraint in _inner_constraints:
target = get_proper_type(constraint.target)
inner_constraints.append(
Constraint(
constraint.original_type_var,
constraint.op,
(
target.copy_modified(last_known_value=None)
if isinstance(target, Instance)
else target
),
)
)
Copy link
Contributor Author

@randolf-scholz randolf-scholz Jul 9, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This hack is needed for testLiteralAndGenericWithUnion and testLiteralMappingContext.

The only purpose is to convert constraints like T ≷ Literal['foo"]? to T ≷ str, which, in these test cases, causes the joint solution to fail due to incompatible constraints like T <: Literal["foo"] and T :> str, hence the longer route of using the outer solution first is used.

@randolf-scholz randolf-scholz mentioned this pull request Jul 9, 2025
Open
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 9, 2025

Diff from mypy_primer, showing the effect of this PR on open source code:

colour (https://github.com/colour-science/colour)
+ colour/utilities/array.py:2428: error: Argument 1 to "__call__" of "_FloatOp" has incompatible type "floating[_16Bit]"; expected "integer[_32Bit] | floating[_32Bit]"  [arg-type]
+ colour/utilities/array.py:2428: error: Argument 1 to "__call__" of "_FloatOp" has incompatible type "floating[_32Bit]"; expected "integer[_16Bit] | floating[_16Bit]"  [arg-type]
+ colour/utilities/array.py:2428: note: Both left and right operands are unions
+ colour/plotting/section.py:390: error: Argument 1 to "LineCollection" has incompatible type "ndarray[tuple[Any, ...], dtype[Any]]"; expected "Sequence[Buffer | _SupportsArray[dtype[Any]] | _NestedSequence[_SupportsArray[dtype[Any]]] | complex | bytes | str | _NestedSequence[complex | bytes | str]]"  [arg-type]

pydantic (https://github.com/pydantic/pydantic)
- pydantic/aliases.py:29: error: Incompatible types in assignment (expression has type "list[str]", variable has type "list[int | str]")  [assignment]
- pydantic/aliases.py:29: note: "list" is invariant -- see https://mypy.readthedocs.io/en/stable/common_issues.html#variance
- pydantic/aliases.py:29: note: Consider using "Sequence" instead, which is covariant
- pydantic/aliases.py:29: error: Argument 1 to "list" has incompatible type "tuple[str | int, ...]"; expected "Iterable[str]"  [arg-type]

scikit-learn (https://github.com/scikit-learn/scikit-learn)
- sklearn/linear_model/tests/test_ridge.py:744: error: Unsupported operand types for + ("list[None]" and "list[ABCMeta]")  [operator]

operator (https://github.com/canonical/operator)
- ops/model.py:2937: error: Argument 2 to "_list_recursive" of "Container" has incompatible type "str | Path"; expected "Path"  [arg-type]
- ops/model.py:3020: error: Argument 2 to "_list_recursive" of "Container" has incompatible type "str | Path"; expected "Path"  [arg-type]
- ops/_private/harness.py:4000: error: Argument 3 to "_notice_matches" of "_TestingPebbleClient" has incompatible type "set[NoticeType | str] | None"; expected "set[str] | None"  [arg-type]

pandas (https://github.com/pandas-dev/pandas)
+ pandas/core/arrays/datetimelike.py:2398: error: Unused "type: ignore" comment  [unused-ignore]
+ pandas/core/arrays/sparse/accessor.py:443: error: Incompatible types in assignment (expression has type "ndarray[tuple[Any, ...], dtype[Any]]", variable has type "list[ndarray[tuple[int], dtype[Any]]]")  [assignment]
+ pandas/core/arrays/sparse/accessor.py:444: error: Incompatible types in assignment (expression has type "ndarray[tuple[Any, ...], dtype[Any]]", variable has type "list[Any]")  [assignment]
+ pandas/core/arrays/sparse/accessor.py:445: error: Incompatible types in assignment (expression has type "ndarray[tuple[Any, ...], dtype[Any]]", variable has type "list[Any]")  [assignment]
+ pandas/core/indexes/interval.py:801: error: Incompatible types in assignment (expression has type "ndarray[tuple[Any, ...], dtype[Any]]", variable has type "list[ndarray[tuple[Any, ...], dtype[Any]] | ndarray[tuple[Any, ...], dtype[signedinteger[_32Bit | _64Bit]]]]")  [assignment]
+ pandas/core/groupby/groupby.py:5650: error: Incompatible types in assignment (expression has type "ndarray[tuple[Any, ...], dtype[Any]]", variable has type "list[Any]")  [assignment]
+ pandas/io/formats/csvs.py:141: error: List item 0 has incompatible type "Hashable"; expected "str"  [list-item]
+ pandas/tests/series/accessors/test_dt_accessor.py:441: error: Unused "type: ignore" comment  [unused-ignore]
+ pandas/tests/scalar/timestamp/test_timestamp.py:124: error: Unused "type: ignore" comment  [unused-ignore]
+ pandas/tests/indexes/datetimes/test_scalar_compat.py:141: error: Unused "type: ignore" comment  [unused-ignore]
+ pandas/tests/dtypes/test_missing.py:806: error: Unused "type: ignore" comment  [unused-ignore]
+ pandas/tests/dtypes/test_missing.py:828: error: Unused "type: ignore" comment  [unused-ignore]
+ pandas/conftest.py:1664: error: Unused "type: ignore" comment  [unused-ignore]

prefect (https://github.com/PrefectHQ/prefect)
- src/prefect/utilities/callables.py:579: error: Incompatible types in assignment (expression has type "list[None]", variable has type "list[Optional[expr]]")  [assignment]
- src/prefect/utilities/callables.py:579: note: "list" is invariant -- see https://mypy.readthedocs.io/en/stable/common_issues.html#variance
- src/prefect/utilities/callables.py:579: note: Consider using "Sequence" instead, which is covariant
- src/prefect/utilities/callables.py:581: error: Unsupported operand types for + ("list[None]" and "list[Optional[expr]]")  [operator]
+ src/prefect/client/orchestration/__init__.py:1088: error: Function does not return a value (it only ever returns None)  [func-returns-value]

artigraph (https://github.com/artigraph/artigraph)
+ src/arti/graphs/__init__.py:181: error: Unused "type: ignore" comment  [unused-ignore]

Expression (https://github.com/cognitedata/Expression)
+ expression/collections/block.py:271: error: Incompatible return value type (got "int", expected "_TSourceSum | Literal[0]")  [return-value]
+ expression/collections/block.py:271: error: Argument 1 to "sum" has incompatible type "tuple[_TSourceSum | Literal[0], ...]"; expected "Iterable[bool]"  [arg-type]
+ expression/collections/block.py:934: error: Incompatible return value type (got "int", expected "_TSourceSum | Literal[0]")  [return-value]
+ expression/collections/block.py:934: error: Argument 1 to "sum" has incompatible type "Block[_TSourceSum | Literal[0]]"; expected "Iterable[bool]"  [arg-type]

jax (https://github.com/google/jax)
+ jax/_src/numpy/lax_numpy.py:2224: error: Unsupported operand types for - ("Sequence[int | Any]" and "int")  [operator]
+ jax/_src/numpy/lax_numpy.py:2224: note: Left operand is of type "Sequence[int | Any] | int | Any"

hydra-zen (https://github.com/mit-ll-responsible-ai/hydra-zen)
- src/hydra_zen/wrapper/_implementations.py:1756: error: Unsupported operand types for + ("list[None]" and "list[str]")  [operator]

altair (https://github.com/vega/altair)
+ tools/generate_schema_wrapper.py:488: error: Dict entry 0 has incompatible type "str": "OverridesItem[MethodSchemaGenerator]"; expected "str": "OverridesItem[SchemaGenerator]"  [dict-item]

static-frame (https://github.com/static-frame/static-frame)
+ static_frame/core/store.py:181: error: Argument 1 to "extend" of "list" has incompatible type "Sequence[TLabel]"; expected "Iterable[str]"  [arg-type]
+ static_frame/core/store.py:181: error: Argument 1 to "extend" of "list" has incompatible type "Sequence[TLabel]"; expected "Iterable[tuple[str, ...]]"  [arg-type]
+ static_frame/core/store.py:183: error: Argument 1 to "extend" of "list" has incompatible type "range"; expected "Iterable[str]"  [arg-type]
+ static_frame/core/store.py:183: note: Following member(s) of "range" have conflicts:
+ static_frame/core/store.py:183: note:     Expected:
+ static_frame/core/store.py:183: note:         def __iter__(self) -> Iterator[str]
+ static_frame/core/store.py:183: note:     Got:
+ static_frame/core/store.py:183: note:         def __iter__(self) -> Iterator[int]
+ static_frame/core/store.py:183: error: Argument 1 to "extend" of "list" has incompatible type "range"; expected "Iterable[tuple[str, ...]]"  [arg-type]
+ static_frame/core/store.py:183: note: Following member(s) of "range" have conflicts:
+ static_frame/core/store.py:183: note:     Expected:
+ static_frame/core/store.py:183: note:         def __iter__(self) -> Iterator[tuple[str, ...]]
+ static_frame/core/store.py:183: note:     Got:
+ static_frame/core/store.py:183: note:         def __iter__(self) -> Iterator[int]

steam.py (https://github.com/Gobot1234/steam.py)
+ steam/abc.py:552: error: Value of type variable "AppT" of "FavouriteBadge" cannot be "PartialApp[str | None] | tuple[str, int]"  [type-var]
- steam/state.py:1453: error: Argument 2 to "pop" of "dict" has incompatible type "None"; expected "ClanInvite | GroupInvite"  [arg-type]
+ steam/state.py:1453: error: Incompatible types in assignment (expression has type "ClanInvite | GroupInvite | None", variable has type "ClanInvite | GroupInvite")  [assignment]

core (https://github.com/home-assistant/core)
+ homeassistant/helpers/service.py:982: error: Argument 1 to "HassJob" has incompatible type "Callable[[ServiceCall], Coroutine[Any, Any, ServiceResponse | EntityServiceResponse] | ServiceResponse | EntityServiceResponse | None]"; expected "Callable[[VarArg([ServiceCall] | [ServiceCall]), KwArg([ServiceCall] | [ServiceCall])], Coroutine[Any, Any, ServiceResponse | EntityServiceResponse] | JsonObjectType | EntityServiceResponse | JsonObjectType | EntityServiceResponse | None]"  [arg-type]
+ homeassistant/components/bthome/coordinator.py:48: error: Returning Any from function declared to return "bool"  [no-any-return]
+ homeassistant/components/xiaomi_ble/coordinator.py:70: error: Returning Any from function declared to return "bool"  [no-any-return]
+ homeassistant/components/motioneye/media_source.py:96: error: Unused "type: ignore" comment  [unused-ignore]

scipy-stubs (https://github.com/scipy/scipy-stubs)
+ tests/sparse/test_construct.pyi:125: error: Expression is of type "coo_array[_ScalarT, tuple[int, int]]", not "coo_array[float64, tuple[int, int]]"  [assert-type]
+ tests/sparse/test_construct.pyi:126: error: Expression is of type "csc_array[_ScalarT]", not "csc_array[float64]"  [assert-type]
+ tests/sparse/test_construct.pyi:127: error: Expression is of type "csr_array[_ScalarT, tuple[int, int]]", not "csr_array[float64, tuple[int, int]]"  [assert-type]
+ tests/sparse/test_construct.pyi:128: error: Expression is of type "coo_array[_ScalarT, tuple[int, int]]", not "coo_array[complex128, tuple[int, int]]"  [assert-type]
+ tests/sparse/test_construct.pyi:129: error: Expression is of type "coo_array[_ScalarT, tuple[int, int]]", not "coo_array[complex128, tuple[int, int]]"  [assert-type]
+ tests/sparse/test_construct.pyi:130: error: Expression is of type "coo_array[_ScalarT, tuple[int, int]]", not "coo_array[complex128, tuple[int, int]]"  [assert-type]

spark (https://github.com/apache/spark)
+ python/pyspark/ml/util.py:1166: error: Unused "type: ignore" comment  [unused-ignore]
+ python/pyspark/ml/tuning.py:257: error: Value of type variable "JP" of "_from_java" of "JavaParams" cannot be "Estimator[Any]"  [type-var]
+ python/pyspark/ml/tuning.py:440: error: Value of type variable "RL" of "loadParamsInstance" of "DefaultParamsReader" cannot be "Estimator[Any]"  [type-var]
+ python/pyspark/ml/classification.py:3662: error: Value of type variable "JP" of "_from_java" of "JavaParams" cannot be "Classifier[Any]"  [type-var]
+ python/pyspark/ml/classification.py:3920: error: Value of type variable "JP" of "_from_java" of "JavaParams" cannot be "Classifier[Any]"  [type-var]
+ python/pyspark/ml/classification.py:3976: error: Unused "type: ignore" comment  [unused-ignore]

antidote (https://github.com/Finistere/antidote)
+ tests/lib/interface/test_custom.py:434: error: Unused "type: ignore" comment  [unused-ignore]
+ tests/lib/interface/test_custom.py:437: error: Unused "type: ignore" comment  [unused-ignore]
+ tests/lib/interface/test_custom.py:440: error: Unused "type: ignore" comment  [unused-ignore]
+ tests/lib/interface/test_custom.py:443: error: Unused "type: ignore" comment  [unused-ignore]
+ tests/lib/interface/test_custom.py:446: error: Unused "type: ignore" comment  [unused-ignore]
+ tests/lib/interface/test_custom.py:449: error: Unused "type: ignore" comment  [unused-ignore]

strawberry (https://github.com/strawberry-graphql/strawberry)
+ strawberry/experimental/pydantic/error_type.py:118: error: Value of type variable "T" of "_wrap_dataclass" cannot be "WithStrawberryObjectDefinition"  [type-var]

discord.py (https://github.com/Rapptz/discord.py)
- discord/ext/commands/bot.py:1316: error: Argument 1 to "find" has incompatible type "Callable[[str], bool]"; expected "Callable[[list[str] | str], Any]"  [arg-type]

schemathesis (https://github.com/schemathesis/schemathesis)
+ src/schemathesis/specs/openapi/_hypothesis.py:397: error: Incompatible types in assignment (expression has type "SearchStrategy[dict[str, Any]]", variable has type "SearchStrategy[None]")  [assignment]
- src/schemathesis/specs/openapi/_hypothesis.py:397: error: Argument 1 to "map" of "SearchStrategy" has incompatible type "Callable[[dict[str, Any]], dict[str, Any]]"; expected "Callable[[dict[str, Any]], None]"  [arg-type]
+ src/schemathesis/specs/openapi/_hypothesis.py:399: error: Incompatible types in assignment (expression has type "SearchStrategy[dict[str, Any]]", variable has type "SearchStrategy[None]")  [assignment]
- src/schemathesis/specs/openapi/_hypothesis.py:399: error: Argument 1 to "map" of "SearchStrategy" has incompatible type "Callable[[dict[str, Any]], dict[str, Any]]"; expected "Callable[[None], None]"  [arg-type]
+ src/schemathesis/specs/openapi/_hypothesis.py:399: error: Argument 1 to "map" of "SearchStrategy" has incompatible type "Callable[[dict[str, Any]], dict[str, Any]]"; expected "Callable[[None], dict[str, Any]]"  [arg-type]

xarray (https://github.com/pydata/xarray)
+ xarray/core/dataset.py: note: In member "drop_dims" of class "Dataset":
+ xarray/core/dataset.py:6108: error: Argument 1 to "set" has incompatible type "Frozen[Hashable, int]"; expected "Iterable[str | None]"  [arg-type]
+ xarray/core/dataset.py:6108: note: Left operand is of type "set[str] | set[Hashable]"
+ xarray/core/dataset.py: note: At top level:
+ xarray/core/dataarray.py: note: In function "_infer_coords_and_dims":
+ xarray/core/dataarray.py:163: error: No overload variant of "__setitem__" of "list" matches argument types "int", "Hashable"  [call-overload]
+ xarray/core/dataarray.py:163: note: Possible overload variants:
+ xarray/core/dataarray.py:163: note:     def __setitem__(self, SupportsIndex, str, /) -> None
+ xarray/core/dataarray.py:163: note:     def __setitem__(self, slice[Any, Any, Any], Iterable[str], /) -> None
+ xarray/core/dataarray.py:186: error: Incompatible types in assignment (expression has type "Hashable", variable has type "str")  [assignment]
+ xarray/core/dataarray.py: note: At top level:
+ xarray/tests/test_dataarray.py:423: error: Unused "type: ignore" comment  [unused-ignore]

zulip (https://github.com/zulip/zulip)
+ zproject/config.py:38: error: Argument "fallback" to "get" of "RawConfigParser" has incompatible type "str | None"; expected "None"  [arg-type]

sympy (https://github.com/sympy/sympy)
+ sympy/core/add.py:384: error: Unused "type: ignore" comment  [unused-ignore]

@randolf-scholz
Copy link
Contributor Author

This still isn't quite correct; there are persisting issues with considering the outer context,

for example 
from typing import TypedDict

class A: ...
class B(A): ...

class OverridesItem[T](TypedDict):
    tp: type[T]

d: dict[str, OverridesItem[A]] = {"foo": OverridesItem(tp=B)}

gives

error: Dict entry 0 has incompatible type "str": "OverridesItem[B]"; expected "str": "OverridesItem[A]"  [dict-item]

with the current version of the PR.

It seems that these issues stem from "solving" variables too eagerly.

In the example above, def [T] (*, tp: type[T`501]) -> TypedDict('tmp_b.OverridesItem', {'tp': type[T`501]}) yields the "solution" T=B, although the constraints only require T :> B, which would allow T=A later on. So possibly the solution should be delayed until the interval (lower, upper) collapses to a single point.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
1 participant
@randolf-scholz